Exponential Choices
27 April 2025
I was recently hired to work on a project that involved porting visual novels to consoles. While I can’t write about those works in detail, thank you NDA’s, I did want to get down a few words about testing a route heavy visual novel, and wrangling with exponential numbers.
The scenario is, we’re far enough into development that I know the script interpreter is generally going to do a good job, and will perform like the original did the majority of the time. My goal is to hunt down the edge cases, find the weird routes and choice combos that lead to a lock up, or result in some odd unwanted behaviour. As one person, that’s a daunting task, so I turn to automation.
For clarity, I’m going to refer to choices and options a lot. When I say choice, I’m talking about when the game will present any number of options for the player to choose from. E.g. a choice might present a menu containing three different options. A route will be a complete path from A to B, of which any number of different choices will be made in between.
The plan is to create an AI player that will run on consoles, testing the game for me. The idea is simple, I set a start point, and an end point, e.g. the start and end of a chapter. The AI player will then auto advance through the game, until it reaches a choice. It’ll check if it’s ever reached this choice before in any previous runs, if not, it’ll make a Save State, and then take the first unused option available to it.
Once it reaches the predetermined endpoint it would mark the run as a success, load the most recent save state, and take the next unused option. This is repeated until all the options in the choice have been tested. It would then move one Save State back, and repeat the process, working itself backwards across all potential routes.
A demonstration of this back to front testing in action.
The theory seems sound enough to me. If left to run this would eventually check all possible route combos. Great! I set it up, and test it’s working on a small section of three separate choices, each with five options. It’s at this point I realise how stupid this approach is.
Assuming that each unique route takes an average of one second to complete, you’re looking at one hundred and twenty-five seconds. Just over two minutes. Not so bad, I’ve got two minutes to spare. Except these first three choices cover roughly thirty seconds of gameplay. A full chapter can be upwards of an hour, and contain many more choices. When I did my first test of the whole game it took roughly one to two minutes to complete, even with text and animation skipping enabled etc.
Ahh crap. Maybe it’s not so bad? It might scale better than I think! What if it takes twenty-four hours to test all routes? That’s a lot of time, but for full route coverage it would be worth it. I did some quick maths. Assuming that we have at least ten choices, each with five options, that’s over nine million unique routes. At one second per route that’s over one hundred days; and since my AI player takes up to two minutes per route and has considerably more choices… Well, that’s not viable.
So what’s the solution? I don’t know! Maybe an army of testers or a better more efficient testing speed? What I ended up doing was tweaking the AI player to now pick a random option when it reached a choice. It would still favour options it hadn’t taken before, choosing randomly from the list of least used options, but it no longer stepped back state by state to one hundred percent test each route. It did mean that since it ran a new route completely when reaching the end, I could leave it running overnight on various devices, and fully test a much larger and varied number of routes from beginning to end; instead of how the old AI Player would have handled it, which due to working backwards was frontloading all the testing of choices near the end of a route.
New random routes, played end to end. I've skipped showing the move through the route as it was less important.
Does it cover everything? No, but it tests a lot. Ultimately, this ended up working great for me, and did help track down a variety of bugs! So not a waste of time! It was useful being able to leave it running on a console, while continuing the port work, and checking in every now and then to see if it had locked up.
I would have liked to have learned more about how devs are testing their larger scale visual novels, especially ones filled with many choices and routes. Asking other dev friends, their answer mostly focused on exhaustive personal testing, followed by post release bug fixes for anything missed! My own searches have led to very little writing on the testing for variable heavy visual novels, but it’s likely I just don’t know what to look for! If you do, let me know!